CCTV Workflow - Redux 2024

Important

The updated version of this document can be found here.

Introduction

This document explains the workflow of CCTV project, from data gathering & processing to training, followed by questions.
Make sure to follow the steps in the README here in order to run the scripts mentioned later in the documentation.
The documentation for CCTV is generated using nbdev and is available here. The link can also be found in the about section of the gqc-utility-notebooks repository.

info

It considers the latest workflow of using DB tables instead of CSV files.
This workflow is exemplified using latest DNV data from 2024

danger

If any portion of the processing requires running a notebook from gqc-utility-notebooks as a notebook itself, make a copy of it under UTLITIY/Data/Dataset/Notebooks folder before modifying.

Usage hints

When running the processing steps on larger datasets, the runtime can be really long. Jake had shared a really useful trick to run time consuming commands through VPN on MSI machine without worrying about loosing the session, using tmux

Jake: Apparently there was a misunderstanding that you needed to run commands directly on MSI to keep them from closing when the SSH or terminal session was closed (such as when connected via VPN SSH on a laptop). This is true unless we use a session tool such as tmux.

 Open a tmux session with "tmux".
 We can start the long-running task on the MSI.
 Then press "Ctrl + b d" to detach from the session.
 You can then close the window.
 You can reattach to the session with "tmux a".

 https://tmuxcheatsheet.com/

Current Workflow

Dataflow diagram

CCTV package is here. Generate the documentation following the instructions in the README file on the package to see more details on how to run each of the following stages.

Video List creation and update
Extracting metadata from metadata DBs
danger
This module will only work on Windows as it requires reading from Access DBs which is only supported under windows.
Import supplementary data
Frame Extraction
Stitch frames and run through Azure OCR
Blurring frames
Extracting distance values from Frames
Extracting condition codes from Frames
SD1_label_images
Training Data Splitter
info
Frame Based (FB) or Video Based (VB) splitting needs to be selected in this step. Set the DROP_ND with the portion ND frames should take from the full dataset. By default it is at 0.5.
These csv files and the extracted blurred_frames.zip containing the corresponding image files are passed on for ML Training section.

Compact View (Needs to be updated)

List of Steps

Data (videos and access DBs) is uploaded to Google drive on hydrotrekai@gqc.com, in this case the SD1 CCTV May 26 2023 upload or SD1D.

The folder structure of the data directory is below:

── CCTV
     └── UTILITY_X
         ├──  Data
         |    └── Dataset_X
         |        ├── Blurred_Frames (has ZIPs)
         |        ├── Extracted_Frames (has ZIPs)
         |        ├── Logs
         |        ├── Received_Data
         |        ├── Temp
         |        ├── Video_DB
         |        |     └── Video_DB.db
         |        |           ├── videos
         |        |           ├── distance_bounding_boxes
         |        |           ├── video-groups-1_frames
         |        |           ├── video-groups-2_frames
         |        |           ├── video_defect_type
         |        |           └── condition_standard
         |        |           └── all_conditions
         |        |           └── labelled_frames
         |        ├── Video_Lists
         |        └── cctv_settings.yml
         └──  scripts (optional)

The Videos uploaded to Google Drive are downloaded using rclone to the Recieved_Data folder.
We can start by manually performing Distance Region extraction and Defect type annotations on videos and save the results to a DB called Video_DB.db under Video_DB folder.
Thus, the DB has distance_bounding_boxes and video_defect_type after above step.
- Note: These are independent and can be done parallelly alongside intial steps mentioned in #8.
Creation and testing RegEx expressions for testing video types based on utility. This is being done manually and is dependent on the video_defect_type

Meanwhile, we run the metadata_DB_extract_inspection_condition_data on the access DB tables to get a CSV called all-conditions.csv. This is also independent for intial steps in #8 and is valid for SD1.

The schema of all_conditions.csv is below, and might need to be altered a bit (reminder that this is based on SD1D)

    | all_conditions.csv             |
    |------------------------------- |
    | video_file                     |
    | video_location                 |
    | distance                       |
    | counter                        |
    | Code                           |
    | continuous_defect              |

Start the CCTV usage notebooks run on the videos under Received_Data folder as follows:
1. Setup YAML parameters with necessary parameters like dataset, utility etc. and provide the path to YAML file while running following scripts.
2. Form video groups using video_list_creator script and save the output to video_groups table.
3. Perform Frame extraction based on the video groups.
  - Using frame_extractor script, the resulting frames are saved to Extracted_Frames folder, with video-group as sub folder name, as ZIPs.
  - Corresponding data is saved to respective tables in format video-group-x_frames. The fields in the video_group_x_frames tables are below:
    TABLE "video-group-1_frames" ( "video_id" : TEXT, "frame_id" : TEXT, "condition_code_list" : TEXT, "OCR_distance" : FLOAT, "distance" : FLOAT, "distance_LF" : TEXT, "OCR_JSON" : JSON, PRIMARY KEY("frame_id") );
4. Run stitch_and_ocr script to generate stitched frames and send to the Azure OCR server. The resulting output is populated in the OCR columns in the video_group_x_frames tables.
5. Then we use blur_frames script to create blurred frames ZIPs for each video group, saved in the Blurred_Frames folder, with video-group as sub folder name.
6. Then we extract distances using extract_distance_values script and populate the video_group_x_frames tables.
7. Using the video_defect_type table, we then extract condition data by running extract_condition_codes script, which populates the condition_code_list in video_group_x_frames tables.
After the above step, we have video_group_x_frames tables with required data to run the final step before training i.e; labelling the blurred frames.
Use create_SD1_label_images script to generate labels for frames using all_conditions table and frames table (specific for the video-group that is being run) from the DB.
The output is stored in database as labelled_frames table. Additional details are in the Questions section.
Then prepare_cctv_training_csv.ipynb needs to be used for creating training, validation and test data.
1. Use split_data_into_train_and_test or split_data_into_train_and_test_VB to separate training+validation set and test set.
2. Then use k_fold_split function to separate the train and validation set. (Provides me with 4 different combinations)
3. Use one of the outputs from 2(b) to train a model.
4. Use the output for test set from 2(a) for testing the model.
Once we are done labelling, we can proceed to compute-msi for training.
1. It require the blurred frames to be copied into train directory.
2. And the train+validation csv generated in above step needs to be used for training.
3. Readme of the repo should contain additional information.

Questions

The below responses were given by Vannary through email.

Label extraction:

Will this notebook, as is, work on SD1 data (or specifically May 26, 2023, upload)? Provided we have distance and defect annotations along with a "all_conditions.csv" generated by grouping access DB files (You can find the details about this csv in the file attached).
- This notebook should work on SD1 data. However, you will have to decide whether you want to label the images according to:
  1. video annotation only (found in the 'labelAbbr' column after you ran the cctv-extract-defect_labels-and-distance-values notebook).
  2. access data/condition data only (this will rely on the distance column and the labels from your condition data csv).
  3. both.
- In addition, you will have to adjust the dist_before, dist_after, buffer_dist, and threshold parameters when using the assign_label_high_spatial_correlation function for each video type.
What are the files we need before we run this notebook? What are the columns in each input file?
- Blurred Frames zip files (this is needed for spatial correlation) (stored in zip file)
- Filtered_Extracted_data.csv (output csv file after running colab-cctv-distance-value-post-processing notebook)
  - columns: 'distance', 'fname_id', 'labelAbbr' (this is only needed for point 1.1)
  - Output is stored in zip file
- condition_data.csv
  - columns: 'cross_reference_id', 'Distance', 'Continuous', 'Code'
  - cross_reference_id - named as 'video_file' in code
- List of videos in the video group (this is optional if you want create image label according to different video type)
  - columns: 'videofile', 'video_type', 'cross_reference_id' (need to match with condition_data.csv).
What is the column structure of the output file?
- Details are included in above answer
If available, can you provide me the list of processing steps we are doing on the input files as part of the notebook above?
- Processing steps:
  - Define a list of video groups you want to process in list_video_groups.
  - Read the labels csv zip file corresponding to each video group (obtained from the colab-cctv-distance-value-post-processing notebook)
  - Read the blurred image zip file corresponding to each video group
  - Find the condition data corresponding to each video using the cross_reference_id column from the condition data csv and the list of videos csv
  - From the condition data, create a list of point defects and a list of continuous defects (note: you can adjust the distance of point defects if needed)
  - For each video in the video group, read the labels csv from the labels csv zip.
  - choose which assign_label_high_spatial_correlation* function to use
  - Drop images labelled as MSA (survey is abandoned) and MCU (camera underwater)
  - Assign unlabelled images as ND
  - Save the dataframe to csv

Steps to edit the labelling notebook

Identifying where CSV files are being imported
- distance labels csv
- the condition data stored in metadata csv
Import SQLite modules and YAML file
Change the path to CSV files to access tables from DB, the Path to DB can be attained from YAML file
Change the column names to names present in the DB tables
Save the output to a new table in the DB, name same as CSV file in code
1. Since we label frames based on groups, should we update existing video-group-X-frames table to have label column in it OR should we create a new table?
2. If creating a new table, we have video_groups to deal with. Should we
  - create a labelled_frames table for each group
  - Maintain a single labelled_frames table with a column 'video_groups' to segregate them

Changes to do

info

The discussion is still in progress

table: video_groups -> videos
Create video_group table with 1 column video_group and refer that table in videos table, using video_group as foreign key
Look into why frames tables are seperated based on groups and not have 1 single file
Look into ways to use video_groups and video_groups_to_skip lists in YAML, especially when the list is large
Add column in pacp_code table to have details on access points instaed of declaring in NB
Verify casing for table names and column names, along with scripts naming.

CCTV Workflow - Redux 2024

Important​

Introduction​

Usage hints​

Current Workflow​

Dataflow diagram​

Compact View (Needs to be updated)​

List of Steps​

Questions​

Steps to edit the labelling notebook​

Changes to do​